I like it. The only thing I really disagree with is having to use mut to be able to mutate through &move pointers; it feels like more trouble than it's worth. I always want to be able to turn an &move into an &mut; &move is strictly a superset of &mut; so having that would be weird, imo.

The other thing is in your unresolved questions, you ask about box destructuring. I don't think these have anything to do with each other.

The "rendered" link is broken.

How does this compare with #965, aside from the keyword used? (cc #998)

@ubsan - The mut requirement came out of an IRC conversation, see example:

let v = Vec::new();
let v2 = &move v;
v2.push(123);

As for box de-structuring... it's probably unrelated, just was a similar case of special-cased Box

@kennytm Fixed the link, will read those issues in a bit.

I might just be too new to the language to see what this solves. I'd love to see a simple but meaningful example of what I can express that I couldn't express before. The examples in the RFC aren't explaining why I want it in a way that I can get it.

I'm not familiar with the difficulty that arises with Box<FnOnce>. Other than that, I see a thing that works mostly like &mut. Why wouldn't I just use regular moves and/or Box?

Partially this is curiosity. But other people will ask this question if the RFC is approved, so I feel justified in asking it now.

How do you avoid the unsafety of deallocate? At the moment, the trait is safe to implement, and yet the deallocate implementation has access to dropped data.

@camlorn With this, you can now owned iterate over an array, for example.

@ubsan What do you mean by "owned iterate over an array", and how does it differ from into_iter?

@bstrie for x in [a, b, c] { drop(x); }

How exactly would this RFC allow for iterating over an owned array?

IntoIterator takes self, not &move self, and we can't change that without breaking backwards compatiblity.

For most usecases, it would be better if we allowed passing DSTs by-value (which under the hood would generate the same code as &move). But if we do that, what usecases remain left for a language-level &move?

@dgrunwald impl<T> IntoIterator for &move [T]. &move is strictly more powerful; you can't currently return "by-value" DSTs, but with this, you could.

@thepowersgang Make sure to add a section about *move, and how it will change Unique.

@camlorn
The problem with Box<FnOnce()> is this: you can't call it. FnOnce::call takes self by value, but a trait object such as FnOnce() is unsized, so you get an error when trying to make the call. However, using boxed closures is desirable when you don't know the concrete type, and specifically Box<FnOnce()> for its only-called-once guarantee. Right now we have the unstable FnBox that can paper over this if you need those guarantees (if you look at the source, you'll see that FnBox::call takes self: Box<Self> which works due to compiler magic and moves the whole box, avoiding the error). There are various ways to fix it, but until we decide which one is perfect we'll have the unstable kludge.

@bstrie
Note that [1, 2, 3].into_iter() is still an Iterator<Item=&i32>.

I'm not sure if I buy this as a solution for the Box<FnOnce> issue. It's my understanding that we already have mechanisms in the compiler for having functions which take by-value self take a pointer to self. If we don't have that, then it probably isn't too hard to add. Today, this alleviates the need for pushing a large structure onto the stack whenever we want to consume it by-value. I think it would be possible to extend that to all trait methods consuming self by-value which are not marked explicitly as where Self: Sized. This is equivalent to the &move proposed here, but it doesn't require adding any complexity to the language itself -- I think it's better to have it done under the hood.

@rphmeier Sure Box works in the compiler, but this is about implementing Box in a library on top of lower-level primitives.

@eddyb I don't fundamentally disagree with the concept of a DerefMove trait, I just don't think that it necessarily goes hand-in-hand with an &move type.

The way we move out of Box today consumes the box and returns the value. To me, that implies a trait like this:

trait DerefMove: DerefMut {
    fn deref_move(self) -> Self::Target;
}

with an implementation for Box like

impl<T: Sized> DerefMove for Box<T> {
    fn deref_move(self) -> T {
        let val = ptr::read(&self.0);
        let (size, align) = (mem::size_of::<T>(), mem::align_of::<T>());
        if size > 0 {
            alloc::deallocate(self.0, size, align);
        }
        mem::forget(self) // skip destructor
        val
    }
}

The compiler can special-case this trait like it does Deref and DerefMut, allowing for Box to be implemented as a library type. Anything related to methods on DSTs which consume self can be done under the hood (at trans time) by passing a pointer to self rather than pushing self onto the stack.

Although the signature of DerefMove inherently doesn't allow for moving unsized types, CoerceUnsized<T> will only work if the struct has one and only one unsizeable field. That means the compiler has to be aware of which field is unsized.

We can pass a pointer to self to functions which expect self by-value (trans will make it work)

These two pieces of information together can enable calling by-value self methods on trait objects by

a) marking the unsized field on the pointer type as already dropped (so we don't drop it again)
b) passing the data pointer of the unsized field to the function found in the field's vtable.

This will cause some interactions with destructors for pointer types which interact with the data they wrap. This is why today we don't allow moving out of types which implement Drop. I think with non-zeroing drop or without drop flags that this strategy wouldn't work anyway.

At the same time, I do think that this could be made to work under the hood without any &move type.

@rphmeier The problem with returning by value there is determining when the underlying allocation is released (and allowing a library type to release that memory)

@rphmeier So you'd be replacing the need for DerefMove with a CoerceUnsized-specific hack?
But that only works if the pointer is to T, not SomeWrapper<T>.
You could add another hack to also adjust the pointer, until the last field matches T.
That seems so messy, although I'm biased towards the implementer's perspective.
And still doesn't let you return an owning reference to a sub-field or anything like that.

Oh, by the way; in my opinion, moving out should not deallocate immediately, but at end of scope. This would allow for people to write

{
  let v = vec![1, 2, 3]
  let x = &move v[..];
  x[1] // still allowed, v is not deallocated yet
}
{
  let v = vec![1, 2, 3];
  let x = &move v[..] as *move _;
  unsafe { (*x)[2] } // still allowed, v is not deallocated yet
}

This means there's no confusing rules about lets or anything, and you can guarantee that the thing will get dropped at end of scope. This is useful for something like:

struct Matrix {
  indices: *move [usize],
  buffer: *move [u8],
}

impl DerefMove for Matrix {
  fn deref_move(&mut self) -> &move [u8] {
    &move *self.buffer
  }
  fn deallocate(&mut self) {
    // I know this is silly, but hopefully it gets the point across
    Box::from_raw(indices);
    Box::from_raw(buffer);
  }
}

@ubsan I agree, the deallocation should be where drops would occur atm, which also makes the borrows simpler to work with (they just have to stop being used at some point before the drop/deallocation).

@camlorn You could argue that you could have a *mut Self -> *mut Self::Target DerefMove which also works in conjuction with functions receiving DSTs by value.
That's the minimal requirement, I suppose.
But from there, you can try to add to it, to make it safe.

My experiments in API design led me to this contraption which is pretty close to maximally flexible for my usecases, but not as advanced as @Ericson2314's ideas.
At the same time, I have no idea how to add a surface syntax to that model.

I'm really not sure what we should do here. Hence why this is a RFC.
Should we just start with a very unsafe DerefMove that works with DSTs?

@camlorn To get some intuition, you can think of it like this. (Warning: intuition. Not deeply technical and precise explanation.) The borrow checker is fundamentally concerned with four things: (a) can I read from this? (b) can I write to this? (c) can I move out of this -- deinitialize it? (d) do I need to initialize this -- move into it? When working with function-local variables (and struct fields, etc.) directly, all four of these are available. You can move out of variables, you can re-initialize them, you can read and write, all as long as the borrow checker can verify that there aren't any conflicts. Indirectly, though -- working through a reference -- only two of those are available: read (&) and write (&mut). This is filling in the hole of being able to pass the right to move out of a variable (or field, etc.) around indirectly via a reference. This matters because there are some things you can only do through references, such as working with unsized types (array slices and trait objects -- like Box<FnOnce>).

@eddyb
I'm not sure what you're trying to tell me. My issue here is that no one is sufficiently explaining the problem or why I want this tool in my toolbox, not that I have a problem with the proposed solution or think we should do it another way.

@glaebhoerl
Thanks. I think I follow what it technically does now, though I'm not quite sure I see the use of it. Still, my principle objection is the difficulty in explaining it and the lack of good motivation in the RFC from my perspective, and you've shown that this is unwarranted. I will now step back and allow the smart people who understand the problems debate under the assumption that eventually someone will write a very good example showing the why of it for the book if it's approved.

I've been following the RFCs because it's fun for like 6 months now and this is the first one where I outright couldn't understand it from the RFC text. I only had slight problems with specialization. Consequently I thought it was probably worth being insistent on the this needs explanation part of the discussion.

@camlorn I do admit that some of us have internalized "we need an owning reference to a borrowed place" for years and the first need I can think of is "being able to move Box to a library while keeping its ability to move out of a dereference, and even extend that to work with unsized values", which is probably a bit obscure.

The second that comes to mind, and which probably affects more code in the wild, is having a common interface over by-value iteration of [T; N], Box<[T]> and Vec<T> - that requires a function that takes ownership over the elements of the array/slice/vector ("owned [T]") and returns an iterator (ByValueIter).
But you also have to keep track of the lifetime so that the iterator doesn't outlive the place where you got the owned slice from, so you end up with &'a move [T] -> ByValueIter<'a>.

Now if we had generics over constant integers, you could argue that you don't need of this ownership manipulation, you can just make one by-value iterator for each of those 3 categories of types ([T; N], Box<[T]> and Vec<T>), and have it own the whole allocation.

There are several options in this area of development of Rust's semantics (including my Own contraption) and none of them truly better than all others, is what I'm trying to say here.
It's not clear that the language could benefit from this feature, unless maybe reduced to bare minimums.
It can be hard to explain why you may "want this tool in your toolbox" if it's not clear that you do.

@camlorn #965 has some more examples which may help too.

#965 does help. Everyone else having my problem should go there too. Thanks. I'm starting to see the why of it, but I think I need to see it in practice before I fully grasp the motivation.

One interesting point of the linked issue is that this helps avoid unnecessary copies. Is this still a possible use? If so, I want to go elsewhere and start discussing compiler optimizations with a possible eye to working on this one.

@camlorn thanks again for bringing this issue up. I've agreed 100% but hadn't had the time to leave a comment saying so.

The Ember RFC process has a "how do we teach this" section, which is specifically designed to make sure that this kind of thing is considered, I really like it.

@camlorn yes indeed, the extra copying / allocation is still a problem.

Also, for one more very convoluted example, in https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/context.rs#L27 and https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/context.rs#L57 I have a thread context (only for suspended threads) "owning" (with Unique) the allocated stack, and the closure used to spawn the thread receiving a &mut to that stack. This wrong the thread should own itself when alive, and Unique is of course unsafe (and has no lifetime).

With &move I can give the closure real ownership of the new thread, and when the thread exits, I can safely move the &move into the suspended thread context to show it now owns it. (when the thread suspends not exits, I'd still need to transmute &mut -> &move because the thread assumes it will always wake up even if it is killed and doesn't think its given up ownership.)

Also, the init closure, a DST, is written to the allocated stack not boxed separately, so the FnOnce + &move stuff already mentioned would be nice.

@Ericson2314 pls don't use transmute to turn &mut into &move: unsafe { &move *(ptr as *mut _ as *move _)

@ubsan Well yeah I do a variation on yours for Unique today https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/context.rs#L108, and would do that for &move tomorrow.

Ah much simpler: https://github.com/RustOS-Fork-Holding-Ground/libfringe/blob/params/src/fat_args.rs#L34-L36

After passing a pointer to args for the incoming thread, the incoming thread is responsible for moving them out not the outgoing thread. If I had &move I wouldn't need the forget and *mut cast.

One interesting point of the linked issue is that this helps avoid unnecessary copies. Is this still a possible use?

It certainly is. To fix a stack overflow in one unit test which involves moving large FnOnce closures I had to use pointers/memcpy/forget because LLVM was unable to elide the copies between (inlined!) function calls. &move would solve this.

@Ericson2314 Thanks :), I'm not sure why I even said anything.

@mahkoh and anyone else who can answer:

Going briefly off topic in the interest of saving time, is there any theoretical reason the compiler can't optimize this before LLVM and where should I take the discussion (besides setting up to build Rust on my machine)? I have some vague, ten-thousand-foot view ideas as to how to do it. Sort of. And it could be interesting.

@ubsan well I should of said "cast" rather than "transmute", no use being imprecise when I'm choosing the longer, more obscure, word :)!

Obviously this request has a long history (it's also, to some extent, tied in with wanting to move DST by value). We discussed this some in the @rust-lang/lang meeting but before I write that comment I wanted to just give my personal position at the moment. (I've moved around from time to time.)

First, as a generate note, I think that using & here is not the right approach. All &T types indicate that the T is borrowed, but &move T would indicate that the T is owned. Besides the learning hurdle ("wait...so it's borrowed...but it's owned..."), this semantic mismatch also shows up elsewhere. For example, you can "reborrow" both &T and &mut T references: but it wouldn't make sense to permit "reborrowing" an &move reference. If fact, a &move expression isn't a borrow at all: logically, it's a move, even though the memory stays put. Moreover, I think we should be extremely careful with adding built-in syntax. There is no easier way to make the language feel complex. This is especially true for a pointer type whose uses are relatively advanced and limited.

The points raised in the previous paragraph suggest that using a type like Moved<'a, T> would be a better choice. Still, whenever I've started to dig into this design, I wind up feeling that something just like Moved<'a, T> is likely not enough. The premise of such a type is that there is a memory slot that contains a T, and you own the T but not the slot -- so presumably there is some outer context that is going to cleanup the slot. For a stack variable, that's easy enough; if we wanted to use Moved<'a, T> to handle DerefMove or something though, we'd need a more general cleanup mechanism. This starts to sound like a ShallowDrop trait, and hence a little like the placer protocol -- it's not obvious to me that this shallow drop trait would be safe, for one thing.

And of course you can go further. If you're trying to work with data in the heap, maybe you don't need the lifetime, for example, you just need a way to free the pointer when you're done. This more or less brings you to eddyb's Own designs, which make things more flexible still. The key idea there is that you would not force a lifetime like 'a, but rather a type that indicates where the value resides, and which can thus encapsulate any cleanup required. So you might have Own<T, Stack<'a>> (no cleanup requires, but cannot escape 'a), or Own<T, Heap> (memory owned by a Box in the heap; when done, we must shallow free this box). This is a very powerful design, but also very complex, and at this point is where I start to get worried about the ergonomics and learnability. After all, we're tinkering here with Rust's "crown jewels" -- ownership and borrowing! -- so it's really important that we tread carefully and think things through deeply.

It would be good, I think, to start slower, by assembling the use cases we aim to address. How many of them are concerned with unsafe code aiming to make maximum efficiency? How many of them are things we expect users to use on a common basis? Where is compiler support needed? And so forth.

I've definitely felt the need for features like this (e.g., I think Rayon could benefit in some places). But I also don't see this as the most pressing problem that Rust faces right now, personally. I'd like to accumulate some more examples and take our time to think on it.

OK, now for the more official comment. :) So yes we talked about this in the @rust-lang/lang meeting a bit. The general consensus was that we'd be inclined to postpone this RFC under the (existing) issue, #998, to be revisited at some later point, when more "design bandwidth" is available. Therefore, I'm going to close this RFC. Thanks @thepowersgang!

(Not sure if I should respond here or #998)

@nikomatsakis It is a big extension right at the core of Rust, but I think that's precisely why I believe new syntax is warranted. Moreover, I see big overlap between this (including out pointers), fixing Drop (c.f. eyepatch, by-value), and, perhaps, non-lexical lifetimes too, and I think together these things do constitute a big priority for the language's evolution. edit and add placer constructs to that list of related language features---even more compelling now. edit 2 and also "temporary" moves out of &mut. edit 3 and linear types.

I still haven't taken the time to fully understand @eddyb's proposal, but I am suspicious because it looks like unsafe encapsulation tricks (safe interface, unsafe implementation) when features I mention above are about making both implementation and interface safe---so its apples and oranges basically.

As to ergonomics/learnability my thoughts are: Yes were making the language more complex, but we're also making it more regular. Because nobody is proposing breaking backwards compat, "beginner's Rust" will stay the same. But I think the trade-off of simplicity for regularity will actually benefit programmers moving to the intermediate/advanced stage---like the point where you realize moving out of Box is hard to write yourself but haven't learned its actually impossible to write yourself is really confusing/infuriating.

For example, you can "reborrow" both &T and &mut T references: but it wouldn't make sense to permit "reborrowing" an &move reference.

Actually it does. One is left with scratch pointer which, when written to, yields another &move. The idea is, given a &move, one can move things in and out of the pointed-to location to their hearts content as long as in the end nothing is there. This is related to moving things out of &mut too---including making swap expressible as a normal function.

In fact, with moving temporarily out of &mut allowed, it to would indicate "borrowing the location, owning its contents". In other places I mentioned there are 4 possible non-aliased mut reference types:

(init at beginning of lifetime, uninit at beginning lifetime) ⨯ (init at end of lifetime, uninit at end of lifetime)

All of these fill a "borrowing the location, owning its contents" role, just with the above restrictions on the initialization status at the lifetime boundaries.

This is especially true for a pointer type whose uses are relatively advanced and limited.

Kinda already said this, but I think it's important to take in account "transitive usage". The code that can be made safe with this is together used ubiquitously, even by beginners.

@Ericson2314

(Not sure if I should respond here or #998)

Yeah, I don't know :) probably #998 is better, since I think the primary question here is not details of this RFC, but rather how to prioritize this change. To be honest, I'd personally prefer to discuss on internals.rust-lang.org, since I think GH doesn't manage sprawling comment threads very well (and throw a link onto #998).

Actually it does [make sense to reborrow an &move]. One is left with scratch pointer which, when written to, yields another &move.

Hmm. I suppose that's true. It's not how borrowing of other things works, but it could be fit in I imagine. Interesting! (It's still rather different from other borows, in that it leaves the value in a distinct state from how it began, but I agree you could make it fit better than I thought initially.)

Moreover, I see big overlap between this (including out pointers), fixing Drop (c.f. eyepatch, by-value), and, perhaps, non-lexical lifetimes too, and I think together these things do constitute a big priority for the language's evolution.

I'm not sure that &move would solve all of those distinct issues :), though certainly it is a better fit for "by-value Drop" than by-value (which is, in fact, a bad fit, hence the reason we chose against adopting it, long ago). But I still feel like it's premature to be adopting an RFC at the moment: we need to get a better handle on the use cases we're aiming at with this design, as well as wrestling some with the mental model that would result.

So, I've been thinking a lot about this RFC over the last few weeks, and I've decided that I was wrong to close it. I didn't intend to "shut down" conversation on the topic, but I think that this is how it came across. I do have lots of doubts about introducing &move -- not the core underlying concept, but the actual syntax -- but I think I should have just tried to air those concerns here, rather than trying to move the conversation elsewhere (e.g. an issue). (Separately, I have some frustrations that we need a better way to coalesce conversations that occur across different forums. But anyway.) So I'm sorry about that.

Anyway, I decided to post this comment last night and at that time I had intended to re-open the RFC (or ask for suggestions about where to move the conversation to). But when I awoke this morning I noticed that in the meantime RFC 1646 has been opened, so perhaps I'll just comment there? (Thoughts?)

This RFC was pretty badly specified, and to be honest the new one is far better :) (I was feeling a little unwell when I wrote this). @nikomatsakis Comments on the syntax should probably be raised on the new RFC pull (maybe as a summary of IRC/forum discussion?)